Rule-based augmentation of a physical environment

ABSTRACT

A group of available virtual objects may each have a respective rule set that indicates how a corresponding available virtual object may be used to augment a physical environment. In some examples, the rule sets may be determined based at least in part on sample images that show approved settings or locations for the available virtual objects. To augment a physical space, one or more characteristics of the physical space may be determined, such as contexts, surface characteristics, and others. The rule sets for the available virtual objects may then be compared to the characteristics of the physical space to determine approved virtual objects that are approved for use to augment the physical space. A selected virtual object may then be selected from the approved virtual objects and inserted into a view of the physical space to create an augmented view of the physical space.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to the following application, which ishereby incorporated by reference in its entirety: U.S. patentapplication Ser. No. 16/216,757 filed Dec. 11, 2018, entitled“RULE-BASED AUGMENTATION OF A PHYSICAL ENVIRONMENT”.

BACKGROUND

In recent years, the ability to change and enhance a person's perceptionof an environment has increased rapidly. In particular, one quicklyexpanding area of technology relates to augmented reality, in which aperception of a real-world environment is augmented bycomputer-generated information, such as by inserting computer-generatedgraphics (e.g., one or more virtual objects) into a view of a physicalspace. In some examples, images and/or video of a physical space, suchas a particular room or outdoor setting, may be captured using a camera.An image may then be augmented, such as by inserting one or more virtualobjects into the image to overlay various locations within the image. Insome other examples, a view of a physical space may be augmented byoverlaying one or more virtual objects over a view of a physical spaceitself, such as by displaying the virtual objects on an eyeglass orother translucent display. These and other view augmentationtechnologies may serve a wide variety of applications, such as gaming,training, education, industrial design, interior decorating and design,real estate and architectural design and many others.

BRIEF DESCRIPTION OF DRAWINGS

The following detailed description may be better understood when read inconjunction with the appended drawings. For the purposes ofillustration, there are shown in the drawings example embodiments ofvarious aspects of the disclosure; however, the invention is not limitedto the specific methods and instrumentalities disclosed.

FIG. 1 is a diagram illustrating an example rule-based view augmentationsystem that may be used in accordance with the present disclosure.

FIG. 2 is a diagram illustrating an example sample image-based rule setdetermination technique that may be used in accordance with the presentdisclosure.

FIG. 3 is a diagram illustrating an example physical spacecharacteristic determination technique that may be used in accordancewith the present disclosure.

FIG. 4 is a diagram illustrating an example rule set comparison that maybe used in accordance with the present disclosure.

FIG. 5 is a diagram illustrating example rule-based filtering and objectselection that may be used in accordance with the present disclosure.

FIG. 6 is a flowchart illustrating an example process for sampleimage-based rule set determination that may be used in accordance withthe present disclosure.

FIG. 7 is a flowchart illustrating an example process for rule-basedaugmentation of a physical environment that may be used in accordancewith the present disclosure.

FIG. 8 is a diagram illustrating an example system for transmitting andproviding data that may be used in accordance with the presentdisclosure.

FIG. 9 is a diagram illustrating an example computing system that may beused in accordance with the present disclosure.

DETAILED DESCRIPTION

Various techniques for rule-based augmentation of a physical environmentare described herein. In particular, in some examples, a serviceprovider may receive data associated with virtual objects from thirdparties, such as vendors. In some cases, the received data may be usedto generate computer graphics information for rendering the virtualobjects. As described in detail below, the service provider may providevarious services relating to augmented reality technology. Inparticular, the service provider may facilitate use of the virtualobjects to augment views of a physical space, such as in an augmentedreality context. In some examples, vendors may provide informationindicating how they want the virtual objects to be used in the augmentedreality context. For example, a vendor may provide informationindicating characteristics of an approved location at which the virtualobject may be inserted or rendered in an augmented reality view. In onespecific example, a vendor may provide one or more sample images thatshow a physical object that is similar or identical to, or thatotherwise represents or corresponds to, the virtual object. The sampleimages may depict sample settings or locations at which the vendorapproves of using the physical object, which in turn may be used todetermine approved settings or locations at which the correspondingvirtual object may be used.

In some examples, the service provider may use information from a vendorto determine a rule set for a respective virtual object. The rule setmay include one or more rules for using the respective virtual object inthe augmented reality context. For example, one or more of the rules mayindicate an approved context for the virtual object (e.g., kitchen, den,classroom, office, sporting event, party, outdoors, etc.). One or morerules may also indicate an approved surface for the virtual object(e.g., surface type, surface size, surface shape, surface orientation,surface height, etc.). For example, a rule may indicate that a virtualobject should be placed on a highest detected horizontal surface havingat least a particular size (e.g. 5×10 inches, etc.), shape (e.g.,rectangular, circular, etc.), and/or type (e.g., table, desk, bookshelf,etc.) in a particular room (e.g., kitchen, den, office, etc.). One ormore rules may also indicate approved relationships to other objects ina physical space. For example, a rule may indicate that a virtual objectshould, or should not, be depicted within a particular distance of oneor more other particular physical or virtual objects. As a specificexample, a rule may indicate that a virtual object may be placed on aclosest horizontal plane to any detected currency (e.g., dollar bill orcredit card) in a room. Another rule may indicate that a virtual objectmay not be placed within 5 meters of a television, door and/or window.

Upon being determined, the rule sets may be provided to a rules engine,for example that compares the rule sets to characteristics of a physicalspace in order to assist in augmentation of the physical space with thevirtual objects. In particular, in some examples, data representative ofa physical space may be received. For example, a camera may be used tocapture one or more images and/or other data of a physical space. Thereceived data may then be analyzed to determine one or morecharacteristics of the physical space. For example, in some cases, theimages and/or data may be provided to an object recognition service,such as may be operated by the service provider. The object recognitionservice may analyze the images and/or data to recognize various objectswithin the physical space, such as faces, furniture, doors, windows,pictures, appliances, buildings, vehicles, elements of nature (trees,rivers, clouds, etc.), and the like. Based at least in part on therecognized objects, one or more contexts associated with the physicalspace may then be determined. For example, it may be determined that aphysical space including a refrigerator and a sink is associated with akitchen context, that a physical space including a television and acouch is associated with a den context, and that a physical space with acomputer, desk and chair is associated with an office context. Asanother example, images and/or data may be analyzed using one or moresurface detection techniques to detect locations and characteristics ofsurfaces within a physical space. For example, characteristics such as alocation, type, size, shape, orientation, and height of one or moresurfaces in the physical space may be determined. As yet anotherexample, images and/or data may be analyzed using one or more distanceand/or scaling detection techniques to determine distances betweenvarious objects in the physical space, as well as object depths,heights, and other distances or measurements.

Upon determining the characteristics of a physical space, thecharacteristics may be provided to the rules engine, which may comparethe characteristics to the rule sets of the virtual objects to determinewhich virtual objects are suitable for various locations within thephysical space. For example, using object recognition techniques, it maybe determined that a physical space includes a television and acouch—and that the physical space therefore is a den. Also, usingsurface detection techniques, a particular surface may be identifiedwithin the physical space, and the surface may be determined to have anarea of 5×5 feet. It may also be determined that the surface is 3 feetfrom the couch. These characteristics may then be compared to the rulesets to determine a group of approved virtual objects havingcorresponding rules sets that are satisfied by the characteristics ofthe surface. In some examples, indications of the virtual objects thatare approved for the surface may be provided to a user. For example, insome cases, it may be determined that a virtual coffee mug and a virtualflowerpot have been approved for placement on the surface. The user maythen be informed that the virtual coffee mug and the virtual flowerpotare approved, and the user may be permitted to select the virtual coffeemug and/or the virtual flowerpot for placement on the surface. Uponbeing selected for placement on the surface, the virtual coffee mugand/or virtual flowerpot may be inserted into a view of the physicalspace, such as by being rendered into an image of the physical space, bybeing rendered onto a translucent display (e.g., eyeglasses) throughwhich the physical space is viewed, or by other means. By contrast,other virtual objects whose rule sets are not satisfied by thecharacteristics of the surface may not be approved for surface, and theuser may not be permitted to select those objects for placement on thesurface. For example, a virtual speaker may be approved only forsurfaces that have an area greater than 6×6 feet—and may therefore notbe approved for the surface, which is only 5×5 feet. As another example,a virtual toaster oven may be approved only for placement in akitchen—and may therefore not be approved for the surface, which is in aden instead of a kitchen. As yet another example, a virtual lightingfixture may be approved only for surfaces that are within 2 feet from acouch—and may therefore not be approved for the surface, which is 3 feetaway from the couch.

The ability to provide rule-based filtering of virtual objects foraugmenting a view of a physical space, as described herein, may providea number of advantages. In particular, the rule sets may allow vendorsto have greater control and flexibility over the use of their associatedvirtual objects, such as the ability to control within whichenvironments their associated objects are used. For example, the rulesets may allow vendors to ensure that their virtual objects are not usedin environments that would reduce the perceived value or quality or ofthe objects—or would make the objects appear not to work or functionproperly. For example, rendering a virtual object at a location that istoo big or too small for the object may distort the appearance of theobject (e.g., by making the object appear overly tiny or bulky orotherwise disproportional), make the object appear to not work properly(e.g., by obscuring or distorting the appearance of various parts,components, etc.), may block or distort the surrounding location, or mayotherwise degrade or interfere with the perception of the object and theenvironment. Moreover, the rule-based filtering techniques may allowvendors to prevent their virtual objects from being used in associationwith other objects that may alter or harm the perceived functionality orreputation of the objects. For example, a rule set for a virtualcigarette pack could ensure that the cigarette pack is rendered only inadult contexts, such as bars or casinos or near beer cans, slot machinesor ashtrays—and not in child-related contexts, such as classrooms,playgrounds, and the like. Additionally, the rule-based techniquesdescribed herein may improve efficiency and reliability for users, suchas by selecting, for the users, a group of virtual objects that areapproved and suitable for a particular location, while filtering outvirtual objects that are disapproved and unsuitable for a particularvirtual location. This may save users from the time required to attemptto manually insert a virtual object into a particular location on anobject-by-object basis, and to manually determine that the virtualobject is too big or too small or otherwise unsuitable for the location.This may also reduce processing time and occurrence of errors related torendering virtual objects that are too big, too small or otherwiseunsuitable for a particular location.

Referring now to FIG. 1, an example rule-based view augmentation systemwill now be described in detail. In the example of FIG. 1, vendors 100may provide data associated with available virtual objects 111 toservice provider 110. The available virtual objects 111 may, forexample, include computer generated graphics that represent or depict ofvarious products, consumer goods, people or animals, structures,elements of nature, or other items or objects. In some examples, theavailable virtual objects 111 may include graphics that representphysical objects that are sold or distributed by the vendors 100.However, there is no requirement that the vendors 100 must sell ordistribute any such physical objects. There is also no requirement thatany or all of available virtual objects 111 must necessarily represent acorresponding physical object that exists in the real world. In someexamples, the vendors 100 may provide computer graphics information forrendering the available virtual objects 111. In other examples, thevendors 100 may provide sample images of physical objects that arerepresented by the available virtual objects 111, and the serviceprovider 110 may generate computer graphics information for renderingthe virtual objects based on the provided sample images. Various othertechniques may also be employed for generating the available virtualobjects 111, such as based on descriptions of the available virtualobjects or other data and/or metadata.

As also shown in FIG. 1, rule sets 105 may include rules for using theavailable virtual objects 111 in the augmented reality context. In someexamples, each available virtual object 111 may have a respective ruleset 105. The respective rule set 105 may include rules indicatingcharacteristics of a suitable physical space in which the correspondingavailable virtual object 111 is approved for use. For example, one ormore of the rules may indicate an approved context for the virtualobject (e.g., kitchen, den, classroom, office, sporting event, party,outdoors, etc.). One or more rules may also indicate an approved surfacefor the virtual object (e.g., surface type, surface size, surface shape,surface orientation, surface height, etc.). For example, a rule mayindicate that a virtual object should be placed on a highest detectedhorizontal surface having at least a particular size (e.g. 5×10 inches,etc.), shape (e.g., rectangular, circular, etc.), and/or type (e.g.,table, desk, bookshelf, etc.) in a particular room (e.g., kitchen, den,office, etc.). One or more rules may also indicate approvedrelationships to other objects in a physical space. For example, a rulemay indicate that a virtual object should, or should not, be depictedwithin a particular distance of one or more other particular physical orvirtual objects. As a specific example, a rule may indicate that avirtual object may be placed on a closest horizontal plane to anydetected currency (e.g., dollar bill or credit card) in a room. Anotherrule may indicate that a virtual object may not be placed within 5meters of a television, door and/or window.

The rule sets 105 may be determined using a variety of differenttechniques. Referring now to FIG. 2, one example rule set determinationtechnique will now be described in detail. In the example of FIG. 2, avendor 200 provides one or more sample images 220 that show a physicalobject that is similar or identical to, or that otherwise represents orcorresponds to, the virtual object. The sample images may depict samplesettings or locations at which the vendor approves of using the physicalobject, which in turn may be used to determine approved settings orlocations at which the corresponding virtual object may be used. Inparticular, as shown in FIG. 2, the sample images 220 may be provided toimage analysis components 230, which may include one or morecomputer-executed components, such as may be used to determine sampleimage characteristics 240, such as characteristics of the locations andsettings of the physical object in the sample images 220. In someexamples, the image analysis components 230 may employ various objectrecognition techniques to recognize the physical object as well as oneor more objects that surround the physical object in each of the sampleimages 220. In some examples, the physical object may be recognized inthe sample images 220 based at least in part on the visualcharacteristics of the respective virtual object to which itcorresponds. Additionally, in some examples, the object recognitiontechniques may be employed to recognize and classify objects thatsurround the physical object in the sample images 220. In some examples,this may allow an approved context for the respective virtual object tobe determined. For example, if the physical object is surrounded by acouch and a television in one or more sample images 220, then it may bedetermined that the physical object is located in a den—and that a denis therefore an approved context for the virtual object to which thephysical object corresponds. As another example, if the physical objectis surrounded by a refrigerator and a sink in one or more sample images220, then it may be determined that the physical object is located in akitchen—and that a kitchen is therefore an approved context for thevirtual object to which the physical object corresponds. Additionally,in some examples, one or more surface detection and/or scalingtechniques may be employed to determine characteristics of the physicalobject (e.g., size, shape, etc.) as well as characteristics of surfaceson which the physical object is positioned in the sample images 220,such as surface type, surface size, surface shape, surface orientation,surface height, etc. These may be used to determine characteristics ofapproved surfaces for the virtual object to which the physical objectcorresponds. Furthermore, in some examples, distances and otherrelationships may be determined between the physical object and otherobjects in the sample images 220, and these may be used to determineapproved distances and other relationships to other objects for thevirtual object to which the physical object corresponds. For example, ifit is determined that the physical object is consistently positionedwithin 2 feet of a couch in the sample images 220, then it may bedetermined that the respective virtual object should be rendered toappear to be located within 2 feet of a couch in an augmented physicalspace.

As shown in FIG. 2, the sample image characteristics 240 may be used todetermine a rule set 250 for the respective virtual object, such as byusing the above described techniques. In some examples, upondetermination of the rule set 250 based on the sample imagecharacteristics 240, the rule set 250 may be provided to the vendor 200for review. During the review, the vendor 200 may confirm one or morerules that meet the vendor's approval and/or may delete or modify one ormore rules that do not meet the vendor's approval. Upon being confirmed,the rule set 250 may be made available to the rules engine 112. It isnoted that the sample image analysis process described above and shownin FIG. 2 is merely one example technique for determination of a ruleset 250 and that one or more other techniques may additionally oralternatively be employed. For example, in some cases, the vendors 100may provide a textual or other description of one or more approvedlocations for the virtual objects, and the description may be parsed toidentify various keywords, phrases or other indicators of various rulesfor use of the virtual objects. In yet other examples, the serviceprovider 110 may request information for determination of the rule sets,such as by issuing one or more queries and/or providing one or moreforms, templates or interfaces for entry of rule set information, forexample via drop down menus, buttons, checkboxes, or other inputcontrols. In yet other examples, machine learning algorithms may beemployed to assist in determination of rule sets. For example, in somecases, a particular virtual object may be correlated to other similarvirtual objects having the same visual characteristics, description orother similar qualities as the particular virtual object. The rule setsfor those similar virtual objects may then be applied to the particularvirtual object, such as by using those rule sets as a starting point orbaseline for the rule set of the particular virtual object—andpotentially modifying those rule sets using any or all of the techniquesset forth above.

In some examples, the sample images 220 of FIG. 2 may additionally oralternatively depict sample settings or locations at which the vendor200 disapproves of using a particular virtual object. For example, thesample images 220 may show various contexts (e.g., den, kitchen, outdoorsettings, etc.) at which vendor 200 wishes to prohibit or discourage useof a virtual object. The sample images may also depict scenarios inwhich a physical object is positioned at a surface/position that isunsuitable for a corresponding virtual object (e.g., a surface that istoo large, too small, too high, tool low, too close to one or more otherobjects, too far from one or more other objects, etc.). These and othersample image characteristics 240 may be used to determine one or morecharacteristics of a physical space in which a virtual object may beprohibited or discouraged from being used. Thus, in these and otherexamples, rule set 250 may include one or more rules indicating one ormore characteristics of the physical environment that prohibit use ofthe virtual object to augment the physical environment. For example, arule may indicate that a particular virtual object cannot be used in aden or a kitchen. In other examples, a rule may indicate characteristicsof a surface and/or position in which a particular virtual object cannotbe used, such as a surface that exceeds a maximum size or height, thatis smaller or lower than a minimum size or height, that is too close toanother specified object, that is too far from another specified object,and many others. Thus, a rule set 250 may include rules relating tousing a virtual object to augment a physical environment, which mayinclude rules indicating characteristics of a physical environment atwhich a virtual object is approved for use as well as rules indicatingcharacteristics of a physical environment at which a virtual object isprohibited from use.

Referring back to FIG. 1, it is shown that rule sets 105 may bedetermined for a group of available virtual objects 111. As will now bedescribed in detail, the rule sets 105 may be used to filter theavailable virtual objects 111, by rules engine 112, in order todetermine a sub-group of the available virtual objects, referred tohereinafter as approved virtual objects 120, which are approved for usein association with a particular physical space. For example, FIG. 1includes a physical space 160, which is a real-world three-dimensionalspace, such as room (e.g., kitchen, den, office, classroom, industrialsetting, etc.), outdoor setting (e.g., park, playground, etc.), or otherphysical space. Physical space view 145 is a view of the physical space160, such as an image or video of the physical space 160—or a view ofthe actual physical space 160 itself, for example as may be seen througheyeglasses and/or a translucent display through which the physical space160 is viewed. As will also be described in detail below, selectedvirtual objects 140 may be selected from the approved virtual objects120 and inserted into the physical space view 145, such as to create anaugmented view 150 of the physical space 160, for example in anaugmented reality context. In some examples, the selected virtualobjects 140 may be inserted into one or more images and or video of thephysical space 160. In other examples, the virtual objects may bedisplayed via eyeglasses and/or a translucent display through which thephysical space 160 is viewed.

In the example of FIG. 1, physical space data 135 corresponding tophysical space 160 is captured, such as via one or more cameras, sensorsor other data capture components. In some examples, the physical spacedata 135 may include one or more images, image data, and/or other datathat identifies attributes of the physical space. The physical spacedata 135 may be analyzed, for example by one or more computer-executeddata analysis processes, to determine physical space characteristics125, which are characteristics of the physical space 160. Referring nowto FIG. 3, some example techniques for determination of the physicalspace characteristics 125 will now be described in detail. As shown inFIG. 3, physical space data 135 for physical space 160 may be providedto object recognition components 310, such as a service may be operatedby the service provider 110. The object recognition components 310 mayanalyze physical space data 135 to recognize various objects within thephysical space, such as faces, furniture, doors, windows, pictures,appliances, buildings, vehicles, elements of nature (trees, rivers,clouds, etc.), and the like. Based at least in part on the recognizedobjects, one or more contexts associated with the physical space maythen be determined. For example, it may be determined that a physicalspace including a refrigerator and a sink is associated with a kitchencontext, that a physical space including a television and a couch isassociated with a den context, and that a physical space with acomputer, desk and chair is associated with an office context.

As also shown in FIG. 3, physical space data 135 may be analyzed bysurface detection components 320 to detect locations and characteristicsof surfaces within the physical space 160. For example, surfacecharacteristics such as a location, type, size, shape, orientation, andheight of one or more surfaces in the physical space 160 may bedetermined. Additionally, physical space data 135 may be analyzed byscaling components 330 to determine distances between various objects inthe physical space 160, as well as object depths, heights, and otherdistances or measurements. Furthermore, physical space data 135 may beanalyzed other analysis components 340, such as one or more machinelearning components, to determine physical space characteristics 125.For example, in some cases, a particular physical space includingvarious objects or other characteristics may be correlated to otheranalyzed physical spaces having the same or similar characteristics. Atleast some of the characteristics for those other similar physicalspaces may then be applied to the particular physical space that isbeing analyzed. Thus, in the example of FIG. 3, components 310-340 maybe employed to determine physical space characteristics 125, which maybe provided to rules engine 112.

Referring back to FIG. 1, it is shown that rules engine 112 may performrule-based filtering 113 to select, from the available virtual objects111, a group of approved virtual objects 120 that are approved for useto augment the physical space view 145 of the physical space 160. Aspart of the rule-based filtering 113, the rules engine 112, may comparethe physical space characteristics 125 to the rule sets 105 to determineone or more of the rule sets 105 that are sufficiently satisfied by thephysical space characteristics 125. In some examples, one or more of theavailable virtual objects 111 having respective rule sets 105 that aresufficiently satisfied by the physical space characteristics 125 maythen be included in the approved virtual objects 120. By contrast, oneor more other of the available virtual objects 111 having respectiverule sets 105 that are not sufficiently satisfied by the physical spacecharacteristics 125 may not be included in the approved virtual objects120. It is noted that it is not necessarily required that physical spacecharacteristics 125 must meet each and every requirement of a rule setin order for the rule set to be considered sufficiently satisfied by thephysical space characteristics 125 such that a corresponding virtualobject is included in the approved virtual objects 120. For example, insome cases, it may only be necessary for the physical spacecharacteristics 125 to meet a threshold percentage or number of rulesfor a rule set to be considered sufficiently satisfied by the physicalspace characteristics 125. In other examples, the approved virtualobjects 120 may include virtual objects whose rules sets are not fullysatisfied by the physical space characteristics 125—but that insteadwere closer or closest matches to the physical space characteristics 125in comparison to the rule sets of other available virtual objects 111.In some examples, an indication may be provided to a user of an extentto which a particular virtual object's rule set is satisfied by thephysical space characteristics 125. Also, in some examples, the approvedvirtual objects 120 may be weighted, ranked, or otherwise orderedrelative to one another to indicate a relative extent to which eachvirtual object's rule set is satisfied by the physical spacecharacteristics 125.

An example of rule-based filtering 113 will now be described in detailwith reference to FIG. 4. In particular, in the example of FIG. 4, thephysical space characteristics 125 include object recognition results405 and surface characteristics 410 as well as other characteristics(the various ellipses shown in FIG. 4 indicate that othercharacteristics may also be determined in addition to those specificallyshown in FIG. 4). The object recognition results 405 indicate that, inthis example, the physical space 160 includes a couch, a television anda coffee table, for example as may be detected by object recognitioncomponents 310 of FIG. 3. The object recognition results 405 furtherindicate that, based on these detected objects, the context of physicalspace 160 is determined to be a den. In FIG. 4, the physical spacecharacteristics 125 also include surface characteristics 410, which arecharacteristics for the surface of the detected coffee table within thephysical space (other surface characteristics for other detectedsurfaces may also be determined, but are not shown in FIG. 4). As shown,the surface characteristics 410 indicate that the coffee table surfacehas an area of 5×5 feet, is a horizontal surface, has a height of 2feet, and is positioned 3 feet from the couch and 5 feet from thetelevision.

In the example of FIG. 4, there are five available virtual objects 111having respective rule sets 421-425. In particular, rule set 421corresponds to a virtual coffee mug, rule set 422 corresponds to avirtual speaker, rule set 423 corresponds to a virtual toaster oven,rule set 424 corresponds to a virtual flowerpot, and rule set 425corresponds to a virtual lighting fixture. In this example, each of therule sets 421-425 may be compared to the physical space characteristics125 to determine whether each rule set 421-425 is satisfied by thephysical space characteristics 125. In particular, rule set 421indicates that the virtual coffee mug may be used in both den andkitchen contexts, and that a minimum surface size upon which the virtualcoffee mug can be placed is 1×1 feet. In this example, because thecontext of physical space 160 is a den (as indicated in objectrecognition results 405) and because the coffee table has a surface sizeof 5×5 feet (as indicated in surface characteristics 410) that exceedsthe minimum surface size of 1×1 feet, the rule set 421 is satisfied bythe physical space characteristics 125 for the coffee table surface.Also, in this example, because the rule set 421 is satisfied, thevirtual object corresponding to rule set 421, which is the virtualcoffee mug, is approved for placement on the coffee table surface. FIG.4 displays a status 431 for the virtual coffee mug, which indicates thatthe virtual coffee mug is approved—and is therefore included in theapproved virtual objects 120 for the coffee table surface.

As another example, rule set 422 indicates that the virtual speaker maybe used in a den context, and that a minimum surface size upon which thevirtual speaker can be placed is 6×6 feet. In this example, the contextof physical space 160 is a den, which satisfies the den requirement ofrule set 422. However, the coffee table has a surface size of only 5×5feet, which is less than the minimum surface size of 6×6 feet in ruleset 422. For this reason, the rule set 422 is not satisfied by thephysical space characteristics 125 for the coffee table surface. In thisexample, because the rule set 422 is not satisfied, the virtual objectcorresponding to rule set 422, which is the virtual speaker, is notapproved for placement on the coffee table surface. FIG. 4 displays astatus 432 for the virtual speaker, which indicates that the virtualspeaker is not approved—and is therefore not included in the approvedvirtual objects 120 for the coffee table surface.

As another example, rule set 423 indicates that the virtual toaster ovenmay be used in a kitchen context, and that a minimum surface size uponwhich the virtual speaker can be placed is 3×3 feet. In this example,the coffee table has a surface size of 5×5 feet, which satisfies theminimum surface size of 3×3 feet in rule set 423. However, the contextof physical space 160 is a den, which does not satisfy the kitchencontext requirement of rule set 423. For this reason, the rule set 423is not satisfied by the physical space characteristics 125 for thecoffee table surface. In this example, because the rule set 423 is notsatisfied, the virtual object corresponding to rule set 423, which isthe virtual toaster oven, is not approved for placement on the coffeetable surface. FIG. 4 displays a status 433 for the virtual toasteroven, which indicates that the virtual toaster oven is not approved—andis therefore not included in the approved virtual objects 120 for thecoffee table surface.

As another example, rule set 424 indicates that the virtual flowerpotmay be used in a den context, and that a maximum surface size upon whichthe virtual flowerpot can be placed is 6×6 feet. In this example,because the context of physical space 160 is a den and because thecoffee table has a surface size of 5×5 feet that does not exceed themaximum surface size of 6×6 feet, the rule set 424 is satisfied by thephysical space characteristics 125 for the coffee table surface. Also,in this example, because the rule set 424 is satisfied, the virtualobject corresponding to rule set 424, which is the virtual flowerpot, isapproved for placement on the coffee table surface. FIG. 4 displays astatus 434 for the virtual flowerpot, which indicates that the virtualflowerpot is approved—and is therefore included in the approved virtualobjects 120 for the coffee table surface.

As another example, rule set 425 indicates that the virtual lightingfixture may be used in a den context and that the virtual lightingfixture must be within 2 feet of a couch. In this example, the contextof physical space 160 is a den, which satisfies the den contextrequirement of rule set 425. However, the coffee table surface is 3 feetaway from the couch, which does not satisfy the requirement that thevirtual lighting fixture must be within 2 feet of a couch. For thisreason, the rule set 425 is not satisfied by the physical spacecharacteristics 125 for the coffee table surface. In this example,because the rule set 425 is not satisfied, the virtual objectcorresponding to rule set 425, which is the virtual lighting fixture, isnot approved for placement on the coffee table surface. FIG. 4 displaysa status 435 for the virtual lighting fixture, which indicates that thevirtual lighting fixture is not approved—and is therefore bot includedin the approved virtual objects 120 for the coffee table surface.

Thus, as shown in the example of FIG. 4, physical space characteristics125 may be compared to rule sets 421-425 to determine approved virtualobjects 120 for use in a physical space 160, such as at a particularsurface or other location within the physical space 160 (e.g., thecoffee table surface described above). The comparison of physical spacecharacteristics 125 to rule sets 421-425 may be included as part ofrule-based filtering 113, an example of which will now be described indetail with reference to FIG. 5. In particular, in the example of FIG.5, available virtual objects 111 may include virtual coffee mug 511,virtual speaker 512, virtual toaster oven 513, virtual flowerpot 514,and virtual lighting fixture 515, which correspond to rule sets 421-425of FIG. 4, respectively. As shown in FIG. 5, rule-based filtering 113 isused to determine approved virtual objects 120 from the availablevirtual objects 111. In this example, the approved virtual objects 120are approved for use at the coffee table surface of physical space 160described above. In particular, using the comparison shown in FIG. 4, itis seen that the approved virtual objects 120 include virtual coffee mug511 and virtual flowerpot 514, as indicated by the approved statuses 431and 434 of FIG. 4, respectively. As was also shown in the example ofFIG. 4, virtual speaker 512, virtual toaster oven 513 and virtuallighting fixture 515 are not approved for use with the coffee tablesurface, as indicated by the not approved statuses 432, 433 and 435,respectively, and are therefore not included in approved virtual objects120 in FIG. 5.

Next, as shown in FIGS. 1 and 5, a selector 130, such as a user, mayselect one or more selected virtual objects 140 from the group ofapproved virtual objects 120 for use in augmenting physical space view145 to form augmented view 150. In particular, in the example of FIG. 5,the selector 130 has selected virtual coffee mug 511 for insertion intothe augmented view 150. In this example, because the virtual coffee mug511 is selected for use at the coffee table surface, the virtual coffeemug may be inserted and positioned in the augmented view 150 on thecoffee table surface. It is noted that, while FIGS. 4-5 show an exampleobject selection for the coffee table surface, the processes describedin FIGS. 4-5 may be repeated for other detected surfaces or locationswithin the physical space 160. For example, in some cases, additionalsurfaces could be detected within the physical space 160, such as an endtable surface, a bookshelf surface, an entertainment center surface,etc. The processes of FIGS. 4 and 5 could then be repeated to determineone or more approved virtual objects 120 and selected virtual objects140 to insert and position at those additional surfaces/locations withinthe augmented view 150.

It is further noted that, in addition or as an alternative to beingselected by a user, other selector(s) 130 may also be employed to selectthe selected virtual objects 140 from the approved virtual objects 120.For example, in some cases, the selected virtual objects 140 may bedetermined by assigning various priorities or weights to the virtualobjects, for example with the higher prioritized or higher weightedobjects being selected over the lower prioritized or lower weightedvirtual objects. As set forth above, the priorities or weights may, insome cases, correspond to an extent to which the approved virtualobjects 120 have rule sets that are satisfied by the physical spacecharacteristics 125. Also, in some examples, the priorities or weightsmay be assigned by a computer-executed process, such as a machinelearning analysis, which may for example determine which virtual objectstend to be selected most frequently and/or with higher satisfactionrates at various different contexts, surfaces, locations, etc.

Thus, as described above, rule sets may be used to determine approvedvirtual objects, from which one or more virtual objects may be selectedfor augmentation of a physical space. As also described above, a ruleset for a virtual object may, in some examples, be determined based onsample images of a physical object to which the virtual objectcorresponds. Referring now to FIG. 6, an example sample image-based ruleset determination process will now be described in detail. Inparticular, the process of FIG. 6 is initiated at operation 610, atwhich one or more sample images of a physical object at one or morelocations are received. As set forth above, the sample images may, forexample, be received by a service provider from a vendor. In someexamples, the vendor may offer the physical object for sale or mayotherwise manufacture or provide the physical object. The sample imagesmay depict sample settings or locations at which the vendor approves ofusing the physical object, which in turn may be used to determineapproved settings or locations at which a corresponding virtual object,which represents the physical object, may be used.

At operation 612, one or more characteristics of the one or morelocations is determined based on the one or more sample images. As setforth above, one or more computer-executed image analysis processes maybe performed on the sample images to determine characteristics of thelocations and settings of the physical object in the sample images. Forexample, operation 612 may include performing an object recognitionanalysis to detect the physical object and one or more other objects inthe one or more sample images, such as objects that are adjacent to thephysical object. Additionally, in some examples, the object recognitionanalysis may be employed to recognize and classify the other objects. Insome examples, operation 612 may include determining a context of theone or more locations based, at least in part, on the one or more otherobjects. For example, if the physical object is surrounded by a couchand a television in one or more sample images, then it may be determinedthat the physical object is located in a den. As another example, if thephysical object is surrounded by a refrigerator and a sink in one ormore sample images, then it may be determined that the physical objectis located in a kitchen. Additionally, in some examples, operation 612may include determining a distance or other relationships between thephysical object and the one or more other objects. These may be used todetermine approved distances and other relationships to other objectsfor the virtual object to which the physical object corresponds.Furthermore, in some examples, operation 612 may include determiningcharacteristics of one or more surfaces on which the physical object ispositioned in the one or more sample images, such as surface area, size,shape, orientation, height, type, etc. These may be used to determinecharacteristics of approved surfaces for the virtual object to which thephysical object corresponds.

At operation 614, a rule set is determined for the virtual object thatrepresents the physical object. In particular, the rule set may includeone or more rules for and/or related to using the virtual object toaugment a physical environment. The one or more rules for and/or relatedto using the virtual object to augment a physical environment may begenerated and/or determined based on the one or more firstcharacteristics of the one or more locations determined at operation612. For example, the rule set may include a rule indicating a contextof the physical environment for insertion of the virtual object. Forexample, if the physical object is surrounded by a couch and atelevision in one or more sample images, then it may be determined thatthe physical object is located in a den—and that a den is therefore anapproved context for the virtual object to which the physical objectcorresponds. As another example, if the physical object is surrounded bya refrigerator and a sink in one or more sample images, then it may bedetermined that the physical object is located in a kitchen—and that akitchen is therefore an approved context for the virtual object to whichthe physical object corresponds. The rule set may also include a ruleindicating insertion of the virtual object relative to one or more otherobjects detected within the physical environment. For example, if it isdetermined that the physical object is consistently positioned within 2feet of a couch in the sample images, then it may be determined that therespective virtual object should be rendered to appear to be locatedwithin 2 feet of a couch in an augmented physical space. As anotherexample, the rule set may also include a rule indicating acharacteristic of a surface for insertion of the virtual object withinthe physical environment, such as based on characteristics of detectedsurfaces upon which the physical object was positioned in the sampleimages.

At operation 616, the rule set may be allowed to be optionally modifiedand confirmed, such as by a vendor or other user that provides thesample images. For example, the rules, or indications thereof, may beprovided to the vendor, such as through one or more interfaces. Thevendor 200 may confirm one or more rules that meet the vendor's approvaland/or may delete or modify one or more rules that do not meet thevendor's approval. Additionally, the vendor may also add additionalrules that may not have been determined by the image analysis-basedprocess described above. Upon being confirmed, the rule set may beprovided to the rules engine 112. As set forth above, the rules enginemay be compare the rule set to one or more characteristics of a physicalspace, such as to determine whether the virtual object is approved foruse to augment the physical space.

Thus, rule sets for virtual objects may be determined using techniquessuch as those described above with reference to FIG. 6. An exampleprocess for using rule sets to determine virtual objects that areapproved for use to augment a particular physical space will now bedescribed in detail with reference to FIG. 7. In particular, the processof FIG. 7 is initiated at operation 710, at which physical space data,which is data representative of a physical space, is received. As setforth above, the physical space data may be captured, such as via one ormore cameras, sensors or other data capture components. In someexamples, the physical space data may include one or more images of thephysical space, image data, and/or other data that identifies attributesof the physical space.

At operation 712, one or more characteristics of the physical space aredetermined, for example based at least in part on the physical spacedata. As set forth above, the physical space data may be analyzed, forexample by one or more computer-executed data analysis processes, todetermine the physical space characteristics. In some examples,operation 712 may include determining a type of object in the physicalspace and/or a context of the physical space, such as based at least inpart on an object recognition analysis. For example, an objectrecognition analysis may be performed on the physical space data torecognize various types of objects within the physical space, such asfaces, furniture, doors, windows, pictures, appliances, buildings,vehicles, elements of nature (trees, rivers, clouds, etc.), and thelike. Based at least in part on the recognized objects, one or morecontexts associated with the physical space may then be determined. Forexample, it may be determined that a physical space including arefrigerator and a sink is associated with a kitchen context—or that aphysical space including a television and a couch is associated with aden context. Also, in some examples, operation 712 may includedetermining a characteristic of a surface detected in the physicalspace. For example, a surface detection analysis may be performed on thephysical space data to detect locations and characteristics of surfaceswithin the physical space. Surface characteristics such as a location,type, size, shape, orientation, and height of one or more surfaces inthe physical space may be determined. Additionally, operation 712 mayinclude determining distances between various objects in the physicalspace, as well as object depths, heights, and other distances ormeasurements.

At operation 714, the characteristics of the physical space are comparedto one or more rule sets. For example, as described above, for eachvirtual object in a group of available virtual objects (e.g., availablevirtual objects 111 of FIG. 1), a respective rule set may be determinedthat corresponds to the virtual object and that includes one or morerules for using the virtual object to augment a physical environment. Asdescribed above, the characteristics of the physical space may becompared to the rule sets to determine one or more rule sets that aresufficiently satisfied by the characteristics of the physical space,such as rule sets in which all rules are satisfied by thecharacteristics of the physical space, rule sets in which a thresholdpercentage or other threshold amount of rules are satisfied by thecharacteristics of the physical space, and/or rule sets that determinedto be closest matches to being satisfied by the characteristics of thephysical space (e.g., as compared to other rule sets for the availablevirtual objects). For example, as described above, a rule regarding aphysical space context may be satisfied when the physical spacecharacteristics indicate that the physical space has the same context asthe context specified by the rule. As another example, a rule regardinga surface characteristic for placement of a virtual object may besatisfied when the physical space characteristics indicate that asurface in the physical space satisfies one or more surfacecharacteristics specified by the rule. Other example rule setcomparisons are described in detail above and are not repeated here.

At operation 716, one or more approved virtual objects (e.g., approvedvirtual objects 120 of FIG. 1) are determined. As set forth above, theapproved virtual objects are objects that are approved for use inaugmenting the physical space. As also set forth above, the approvedvirtual objects may be determined based, at least in part, on thecomparing of the physical space characteristics and the rule setsperformed at operation 714. In some examples, the approved virtualobjects may be a group of virtual objects that are determined from aplurality of virtual objects (e.g., the available virtual objects) andthat are associated with the one or more characteristics of the physicalspace. The group of virtual objects may be associated with the one ormore characteristics of the physical space by one or more rulesrespectively associated with each virtual object of the group of virtualobjects. As set forth above, the approved virtual objects may includevirtual objects whose rule sets are sufficiently satisfied by thecharacteristics of the physical space, such as by being fully satisfied,satisfied beyond a threshold amount, being a closest match (e.g.,relative to other rules sets), or by other means. As a specific example,as shown in FIG. 5, the approved virtual objects 120 may include avirtual coffee mug 511 and a virtual flowerpot 514 from the availablevirtual objects 111. By contrast, virtual speaker 512, virtual toasteroven 513 and virtual lighting fixture 515 are not included in approvedvirtual objects 120 in FIG. 5.

At operation 718, an indication of a selected virtual object from theapproved virtual objects may be received. As set forth above, in someexamples, indications of the approved virtual objects may be provided toa user, and the user may select one or more of the approved virtualobjects to be selected virtual objects that are used to augment a viewof the physical space. In some examples, a list of the approved virtualobjects and/or preview images of the approved virtual objects may beprovided to the user. In some examples, a user may click-on or otherwiseselect a particular surface or other location in an image or other viewof the physical space, and a list of virtual objects that are approvedfor the selected surface or location may then be provided to the user,from which the user may select a particular virtual object to beinserted at the selected surface or location. For example, a user couldclick on a coffee table surface in an image of a physical space to seethat a virtual coffee mug 511 and a virtual flowerpot 514 are approvedfor placement on the coffee table surface. In some examples, selectablepreview images of the virtual coffee mug 511 and the virtual flowerpot514 could be displayed on the coffee table surface. The user couldselect the virtual coffee mug 511 for insertion onto the coffee tablesurface. In some examples, this may include selecting a preview image ofthe virtual coffee mug 511 or selecting the virtual coffee mug 511 froma drop down menu or other selectable interface. Thus, based at least inpart on the comparing of operation 714, it may be determined to use aselected virtual object to augment the physical space. For example,determining to use a selected virtual object to augment the physicalspace may include determining approved virtual objects that haverespective rule sets that are satisfied by the one or morecharacteristics of the physical space, providing, to a user, anindication of the approved virtual objects, and receiving, from theuser, a selection of the virtual object from the approved virtualobjects.

At operation 720, the selected virtual object may be inserted into aview of the physical space (e.g., physical space view 145 of FIG. 1) tocreate an augmented view of the physical space (e.g., augmented view 150of FIG. 1). As set forth above, in some examples, the augmented view ofthe physical space may include the selected virtual object overlaid overan image of the physical space. Also, in some examples, the augmentedview of the physical space may include the selected virtual objectoverlaid over the physical space itself, such as via a translucentdisplay.

An example system for transmitting and providing data will now bedescribed in detail. In particular, FIG. 8 illustrates an examplecomputing environment in which the embodiments described herein may beimplemented. FIG. 8 is a diagram schematically illustrating an exampleof a data center 85 that can provide computing resources to users 70 aand 70 b (which may be referred herein singularly as user 70 or in theplural as users 70) via user computers 72 a and 72 b (which may bereferred herein singularly as computer 72 or in the plural as computers72) via a communications network 73. Data center 85 may be configured toprovide computing resources for executing applications on a permanent oran as-needed basis. The computing resources provided by data center 85may include various types of resources, such as gateway resources, loadbalancing resources, routing resources, networking resources, computingresources, volatile and non-volatile memory resources, content deliveryresources, data processing resources, data storage resources, datacommunication resources and the like. Each type of computing resourcemay be available in a number of specific configurations. For example,data processing resources may be available as virtual machine instancesthat may be configured to provide various web services. In addition,combinations of resources may be made available via a network and may beconfigured as one or more web services. The instances may be configuredto execute applications, including web services, such as applicationservices, media services, database services, processing services,gateway services, storage services, routing services, security services,encryption services, load balancing services, application services andthe like. These services may be configurable with set or customapplications and may be configurable in size, execution, cost, latency,type, duration, accessibility and in any other dimension. These webservices may be configured as available infrastructure for one or moreclients and can include one or more applications configured as aplatform or as software for one or more clients. These web services maybe made available via one or more communications protocols. Thesecommunications protocols may include, for example, hypertext transferprotocol (HTTP) or non-HTTP protocols. These communications protocolsmay also include, for example, more reliable transport layer protocols,such as transmission control protocol (TCP), and less reliable transportlayer protocols, such as user datagram protocol (UDP). Data storageresources may include file storage devices, block storage devices andthe like.

Each type or configuration of computing resource may be available indifferent sizes, such as large resources—consisting of many processors,large amounts of memory and/or large storage capacity—and smallresources—consisting of fewer processors, smaller amounts of memoryand/or smaller storage capacity. Customers may choose to allocate anumber of small processing resources as web servers and/or one largeprocessing resource as a database server, for example.

Data center 85 may include servers 76 a and 76 b (which may be referredherein singularly as server 76 or in the plural as servers 76) thatprovide computing resources. These resources may be available as baremetal resources or as virtual machine instances 78 a-d (which may bereferred herein singularly as virtual machine instance 78 or in theplural as virtual machine instances 78).

The availability of virtualization technologies for computing hardwarehas afforded benefits for providing large scale computing resources forcustomers and allowing computing resources to be efficiently andsecurely shared between multiple customers. For example, virtualizationtechnologies may allow a physical computing device to be shared amongmultiple users by providing each user with one or more virtual machineinstances hosted by the physical computing device. A virtual machineinstance may be a software emulation of a particular physical computingsystem that acts as a distinct logical computing system. Such a virtualmachine instance provides isolation among multiple operating systemssharing a given physical computing resource. Furthermore, somevirtualization technologies may provide virtual resources that span oneor more physical resources, such as a single virtual machine instancewith multiple virtual processors that span multiple distinct physicalcomputing systems.

Referring to FIG. 8, communications network 73 may, for example, be apublicly accessible network of linked networks and possibly operated byvarious distinct parties, such as the Internet. In other embodiments,communications network 73 may be a private network, such as a corporateor university network that is wholly or partially inaccessible tonon-privileged users. In still other embodiments, communications network73 may include one or more private networks with access to and/or fromthe Internet.

Communication network 73 may provide access to computers 72. Usercomputers 72 may be computers utilized by users 70 or other customers ofdata center 85. For instance, user computer 72 a or 72 b may be aserver, a desktop or laptop personal computer, a tablet computer, awireless telephone, a personal digital assistant (PDA), an e-bookreader, a game console, a set-top box or any other computing devicecapable of accessing data center 85. User computer 72 a or 72 b mayconnect directly to the Internet (e.g., via a cable modem or a DigitalSubscriber Line (DSL)). Although only two user computers 72 a and 72 bare depicted, it should be appreciated that there may be multiple usercomputers.

User computers 72 may also be utilized to configure aspects of thecomputing resources provided by data center 85. In this regard, datacenter 85 might provide a gateway or web interface through which aspectsof its operation may be configured through the use of a web browserapplication program executing on user computer 72. Alternately, astand-alone application program executing on user computer 72 mightaccess an application programming interface (API) exposed by data center85 for performing the configuration operations. Other mechanisms forconfiguring the operation of various web services available at datacenter 85 might also be utilized.

Servers 76 shown in FIG. 8 may be servers configured appropriately forproviding the computing resources described above and may providecomputing resources for executing one or more web services and/orapplications. In one embodiment, the computing resources may be virtualmachine instances 78. In the example of virtual machine instances, eachof the servers 76 may be configured to execute an instance manager 80 aor 80 b (which may be referred herein singularly as instance manager 80or in the plural as instance managers 80) capable of executing thevirtual machine instances 78. The instance managers 80 may be a virtualmachine monitor (VMM) or another type of program configured to enablethe execution of virtual machine instances 78 on server 76, for example.As discussed above, each of the virtual machine instances 78 may beconfigured to execute all or a portion of an application.

It should be appreciated that although the embodiments disclosed abovediscuss the context of virtual machine instances, other types ofimplementations can be utilized with the concepts and technologiesdisclosed herein. For example, the embodiments disclosed herein mightalso be utilized with computing systems that do not utilize virtualmachine instances.

In the example data center 85 shown in FIG. 8, a router 71 may beutilized to interconnect the servers 76 a and 76 b. Router 71 may alsobe connected to gateway 74, which is connected to communications network73. Router 71 may be connected to one or more load balancers, and aloneor in combination may manage communications within networks in datacenter 85, for example, by forwarding packets or other datacommunications as appropriate based on characteristics of suchcommunications (e.g., header information including source and/ordestination addresses, protocol identifiers, size, processingrequirements, etc.) and/or the characteristics of the private network(e.g., routes based on network topology, etc.). It will be appreciatedthat, for the sake of simplicity, various aspects of the computingsystems and other devices of this example are illustrated withoutshowing certain conventional details. Additional computing systems andother devices may be interconnected in other embodiments and may beinterconnected in different ways.

In the example data center 85 shown in FIG. 8, a server manager 75 isalso employed to at least in part direct various communications to, fromand/or between servers 76 a and 76 b. While FIG. 8 depicts router 71positioned between gateway 74 and server manager 75, this is merely anexemplary configuration. In some cases, for example, server manager 75may be positioned between gateway 74 and router 71. Server manager 75may, in some cases, examine portions of incoming communications fromuser computers 72 to determine one or more appropriate servers 76 toreceive and/or process the incoming communications. Server manager 75may determine appropriate servers to receive and/or process the incomingcommunications based on factors such as an identity, location or otherattributes associated with user computers 72, a nature of a task withwhich the communications are associated, a priority of a task with whichthe communications are associated, a duration of a task with which thecommunications are associated, a size and/or estimated resource usage ofa task with which the communications are associated and many otherfactors. Server manager 75 may, for example, collect or otherwise haveaccess to state information and other information associated withvarious tasks in order to, for example, assist in managingcommunications and other operations associated with such tasks.

It should be appreciated that the network topology illustrated in FIG. 8has been greatly simplified and that many more networks and networkingdevices may be utilized to interconnect the various computing systemsdisclosed herein. These network topologies and devices should beapparent to those skilled in the art.

It should also be appreciated that data center 85 described in FIG. 8 ismerely illustrative and that other implementations might be utilized. Itshould also be appreciated that a server, gateway or other computingdevice may comprise any combination of hardware or software that caninteract and perform the described types of functionality, includingwithout limitation: desktop or other computers, database servers,network storage devices and other network devices, PDAs, tablets,cellphones, wireless phones, pagers, electronic organizers, Internetappliances, television-based systems (e.g., using set top boxes and/orpersonal/digital video recorders) and various other consumer productsthat include appropriate communication capabilities.

In at least some embodiments, a server that implements a portion or allof one or more of the technologies described herein may include acomputer system that includes or is configured to access one or morecomputer-accessible media. FIG. 9 depicts a computer system thatincludes or is configured to access one or more computer-accessiblemedia. In the illustrated embodiment, computing device 15 includes oneor more processors 10 a, 10 b and/or 10 n (which may be referred hereinsingularly as “a processor 10” or in the plural as “the processors 10”)coupled to a system memory 20 via an input/output (I/O) interface 30.Computing device 15 further includes a network interface 40 coupled toI/O interface 30.

In various embodiments, computing device 15 may be a uniprocessor systemincluding one processor 10 or a multiprocessor system including severalprocessors 10 (e.g., two, four, eight or another suitable number).Processors 10 may be any suitable processors capable of executinginstructions. For example, in various embodiments, processors 10 may beembedded processors implementing any of a variety of instruction setarchitectures (ISAs), such as the x86, PowerPC, SPARC or MIPS ISAs orany other suitable ISA. In multiprocessor systems, each of processors 10may commonly, but not necessarily, implement the same ISA.

System memory 20 may be configured to store instructions and dataaccessible by processor(s) 10. In various embodiments, system memory 20may be implemented using any suitable memory technology, such as staticrandom access memory (SRAM), synchronous dynamic RAM (SDRAM),nonvolatile/Flash®-type memory or any other type of memory. In theillustrated embodiment, program instructions and data implementing oneor more desired functions, such as those methods, techniques and datadescribed above, are shown stored within system memory 20 as code 25 anddata 26.

In one embodiment, I/O interface 30 may be configured to coordinate I/Otraffic between processor 10, system memory 20 and any peripherals inthe device, including network interface 40 or other peripheralinterfaces. In some embodiments, I/O interface 30 may perform anynecessary protocol, timing or other data transformations to convert datasignals from one component (e.g., system memory 20) into a formatsuitable for use by another component (e.g., processor 10). In someembodiments, I/O interface 30 may include support for devices attachedthrough various types of peripheral buses, such as a variant of thePeripheral Component Interconnect (PCI) bus standard or the UniversalSerial Bus (USB) standard, for example. In some embodiments, thefunction of I/O interface 30 may be split into two or more separatecomponents, such as a north bridge and a south bridge, for example.Also, in some embodiments some or all of the functionality of I/Ointerface 30, such as an interface to system memory 20, may beincorporated directly into processor 10.

Network interface 40 may be configured to allow data to be exchangedbetween computing device 15 and other device or devices 60 attached to anetwork or networks 50, such as other computer systems or devices, forexample. In various embodiments, network interface 40 may supportcommunication via any suitable wired or wireless general data networks,such as types of Ethernet networks, for example. Additionally, networkinterface 40 may support communication via telecommunications/telephonynetworks, such as analog voice networks or digital fiber communicationsnetworks, via storage area networks such as Fibre Channel SANs (storagearea networks) or via any other suitable type of network and/orprotocol.

In some embodiments, system memory 20 may be one embodiment of acomputer-accessible medium configured to store program instructions anddata as described above for implementing embodiments of thecorresponding methods and apparatus. However, in other embodiments,program instructions and/or data may be received, sent or stored upondifferent types of computer-accessible media. Generally speaking, acomputer-accessible medium may include non-transitory storage media ormemory media, such as magnetic or optical media—e.g., disk or DVD/CDcoupled to computing device 15 via I/O interface 30. A non-transitorycomputer-accessible storage medium may also include any volatile ornon-volatile media, such as RAM (e.g., SDRAM, DDR SDRAM, RDRAM, SRAM,etc.), ROM (read only memory) etc., that may be included in someembodiments of computing device 15 as system memory 20 or another typeof memory. Further, a computer-accessible medium may includetransmission media or signals such as electrical, electromagnetic ordigital signals conveyed via a communication medium, such as a networkand/or a wireless link, such as those that may be implemented vianetwork interface 40.

A network set up by an entity, such as a company or a public sectororganization, to provide one or more web services (such as various typesof cloud-based computing or storage) accessible via the Internet and/orother networks to a distributed set of clients may be termed a providernetwork. Such a provider network may include numerous data centershosting various resource pools, such as collections of physical and/orvirtualized computer servers, storage devices, networking equipment andthe like, needed to implement and distribute the infrastructure and webservices offered by the provider network. The resources may in someembodiments be offered to clients in various units related to the webservice, such as an amount of storage capacity for storage, processingcapability for processing, as instances, as sets of related services andthe like. A virtual computing instance may, for example, comprise one ormore servers with a specified computational capacity (which may bespecified by indicating the type and number of CPUs, the main memorysize and so on) and a specified software stack (e.g., a particularversion of an operating system, which may in turn run on top of ahypervisor).

A compute node, which may be referred to also as a computing node, maybe implemented on a wide variety of computing environments, such ascommodity-hardware computers, virtual machines, web services, computingclusters and computing appliances. Any of these computing devices orenvironments may, for convenience, be described as compute nodes.

A number of different types of computing devices may be used singly orin combination to implement the resources of the provider network indifferent embodiments, for example computer servers, storage devices,network devices and the like. In some embodiments a client or user maybe provided direct access to a resource instance, e.g., by giving a useran administrator login and password. In other embodiments the providernetwork operator may allow clients to specify execution requirements forspecified client applications and schedule execution of the applicationson behalf of the client on execution platforms (such as applicationserver instances, Java′ virtual machines (JVMs), general-purpose orspecial-purpose operating systems, platforms that support variousinterpreted or compiled programming languages such as Ruby, Perl,Python, C, C++ and the like or high-performance computing platforms)suitable for the applications, without, for example, requiring theclient to access an instance or an execution platform directly. A givenexecution platform may utilize one or more resource instances in someimplementations; in other implementations, multiple execution platformsmay be mapped to a single resource instance.

In many environments, operators of provider networks that implementdifferent types of virtualized computing, storage and/or othernetwork-accessible functionality may allow customers to reserve orpurchase access to resources in various resource acquisition modes. Thecomputing resource provider may provide facilities for customers toselect and launch the desired computing resources, deploy applicationcomponents to the computing resources and maintain an applicationexecuting in the environment. In addition, the computing resourceprovider may provide further facilities for the customer to quickly andeasily scale up or scale down the numbers and types of resourcesallocated to the application, either manually or through automaticscaling, as demand for or capacity requirements of the applicationchange. The computing resources provided by the computing resourceprovider may be made available in discrete units, which may be referredto as instances. An instance may represent a physical server hardwareplatform, a virtual machine instance executing on a server or somecombination of the two. Various types and configurations of instancesmay be made available, including different sizes of resources executingdifferent operating systems (OS) and/or hypervisors, and with variousinstalled software applications, runtimes and the like. Instances mayfurther be available in specific availability zones, representing alogical region, a fault tolerant region, a data center or othergeographic location of the underlying computing hardware, for example.Instances may be copied within an availability zone or acrossavailability zones to improve the redundancy of the instance, andinstances may be migrated within a particular availability zone oracross availability zones. As one example, the latency for clientcommunications with a particular server in an availability zone may beless than the latency for client communications with a different server.As such, an instance may be migrated from the higher latency server tothe lower latency server to improve the overall client experience.

In some embodiments the provider network may be organized into aplurality of geographical regions, and each region may include one ormore availability zones. An availability zone (which may also bereferred to as an availability container) in turn may comprise one ormore distinct locations or data centers, configured in such a way thatthe resources in a given availability zone may be isolated or insulatedfrom failures in other availability zones. That is, a failure in oneavailability zone may not be expected to result in a failure in anyother availability zone. Thus, the availability profile of a resourceinstance is intended to be independent of the availability profile of aresource instance in a different availability zone. Clients may be ableto protect their applications from failures at a single location bylaunching multiple application instances in respective availabilityzones. At the same time, in some implementations inexpensive and lowlatency network connectivity may be provided between resource instancesthat reside within the same geographical region (and networktransmissions between resources of the same availability zone may beeven faster).

As set forth above, content may be provided by a content provider to oneor more clients. The term content, as used herein, refers to anypresentable information, and the term content item, as used herein,refers to any collection of any such presentable information. A contentprovider may, for example, provide one or more content providingservices for providing content to clients. The content providingservices may reside on one or more servers. The content providingservices may be scalable to meet the demands of one or more customersand may increase or decrease in capability based on the number and typeof incoming client requests. Portions of content providing services mayalso be migrated to be placed in positions of reduced latency withrequesting clients. For example, the content provider may determine an“edge” of a system or network associated with content providing servicesthat is physically and/or logically closest to a particular client. Thecontent provider may then, for example, “spin-up,” migrate resources orotherwise employ components associated with the determined edge forinteracting with the particular client. Such an edge determinationprocess may, in some cases, provide an efficient technique foridentifying and employing components that are well suited to interactwith a particular client, and may, in some embodiments, reduce thelatency for communications between a content provider and one or moreclients.

In addition, certain methods or process blocks may be omitted in someimplementations. The methods and processes described herein are also notlimited to any particular sequence, and the blocks or states relatingthereto can be performed in other sequences that are appropriate. Forexample, described blocks or states may be performed in an order otherthan that specifically disclosed, or multiple blocks or states may becombined in a single block or state. The example blocks or states may beperformed in serial, in parallel or in some other manner. Blocks orstates may be added to or removed from the disclosed exampleembodiments.

It will also be appreciated that various items are illustrated as beingstored in memory or on storage while being used, and that these items orportions thereof may be transferred between memory and other storagedevices for purposes of memory management and data integrity.Alternatively, in other embodiments some or all of the software modulesand/or systems may execute in memory on another device and communicatewith the illustrated computing systems via inter-computer communication.Furthermore, in some embodiments, some or all of the systems and/ormodules may be implemented or provided in other ways, such as at leastpartially in firmware and/or hardware, including, but not limited to,one or more application-specific integrated circuits (ASICs), standardintegrated circuits, controllers (e.g., by executing appropriateinstructions, and including microcontrollers and/or embeddedcontrollers), field-programmable gate arrays (FPGAs), complexprogrammable logic devices (CPLDs), etc. Some or all of the modules,systems and data structures may also be stored (e.g., as softwareinstructions or structured data) on a computer-readable medium, such asa hard disk, a memory, a network or a portable media article to be readby an appropriate drive or via an appropriate connection. The systems,modules and data structures may also be transmitted as generated datasignals (e.g., as part of a carrier wave or other analog or digitalpropagated signal) on a variety of computer-readable transmission media,including wireless-based and wired/cable-based media, and may take avariety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). Suchcomputer program products may also take other forms in otherembodiments. Accordingly, the present invention may be practiced withother computer system configurations.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements, and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some or all of the elements in the list.

While certain example embodiments have been described, these embodimentshave been presented by way of example only and are not intended to limitthe scope of the inventions disclosed herein. Thus, nothing in theforegoing description is intended to imply that any particular feature,characteristic, step, module or block is necessary or indispensable.Indeed, the novel methods and systems described herein may be embodiedin a variety of other forms; furthermore, various omissions,substitutions and changes in the form of the methods and systemsdescribed herein may be made without departing from the spirit of theinventions disclosed herein. The accompanying claims and theirequivalents are intended to cover such forms or modifications as wouldfall within the scope and spirit of certain of the inventions disclosedherein.

What is claimed is:
 1. A computing system comprising: one or more computer processors; and one or more memories having stored therein instructions that, upon execution by the one or more computer processors, cause the computing system to perform operations comprising: determining a rule set corresponding to a virtual object, the rule set including one or more rules relating to using the virtual object to augment a physical environment; receiving data representative of a physical space; determining one or more characteristics of the physical space; comparing the one or more characteristics of the physical space to the rule set; based at least in part on the comparing, determining to include the virtual object in a group of virtual objects having rule sets satisfied by the characteristics of the physical space; receiving a selection of the virtual object from the group of virtual objects having rule sets satisfied by the characteristics of the physical space to use the virtual object to augment the physical space; and inserting the virtual object into a view of the physical space to create an augmented view of the physical space.
 2. The computing system of claim 1, wherein determining to use the virtual object to augment the physical space comprises: providing, to a user, an indication of the group of virtual objects; and receiving, from the user, the selection of the virtual object from the group of virtual objects.
 3. The computing system of claim 1, wherein the rule set includes a rule indicating a characteristic of a surface for insertion of the virtual object.
 4. The computing system of claim 1, wherein the rule set includes a rule indicating insertion of the virtual object relative to one or more detected objects.
 5. The computing system of claim 1, wherein the rule set includes a rule indicating a physical environment context for insertion of the virtual object.
 6. The computing system of claim 1, wherein the rule sets include a rule indicating that the one or more characteristics of the physical space prohibit use of another virtual object to augment the physical space.
 7. A computer-implemented method comprising: receiving data representative of a physical space; determining one or more characteristics of the physical space; determining, from a plurality of virtual objects, a group of virtual objects having rule sets that are satisfied by the one or more characteristics of the physical space; receiving an indication of a selected virtual object from the group of virtual objects having rule sets that are satisfied by the one or more characteristics of the physical space; and inserting the selected virtual object into a view of the physical space to create an augmented view of the physical space.
 8. The computer-implemented method of claim 7, wherein one or more rules associated with the selected virtual object include a rule indicating a characteristic of a surface for insertion of the selected virtual object.
 9. The computer-implemented method of claim 7, wherein one or more rules associated with the selected virtual object include a rule indicating insertion of the selected virtual object relative to one or more detected objects.
 10. The computer-implemented method of claim 7, wherein the rule sets include a rule indicating that the one or more characteristics of the physical space prohibit use of another virtual object to augment the physical space.
 11. The computer-implemented method of claim 7, wherein one or more rules associated with the selected virtual object include a rule indicating a physical environment context for insertion of the selected virtual object.
 12. The computer-implemented method of claim 7, wherein determining one or more characteristics of the physical space comprises determining a characteristic of a surface detected in the physical space.
 13. The computer-implemented method of claim 7, wherein determining one or more characteristics of the physical space comprises determining a context of the physical space based at least in part on an object recognition analysis.
 14. The computer-implemented method of claim 7, wherein determining one or more characteristics of the physical space comprises detecting a type of an object in the physical space based at least in part on an object recognition analysis.
 15. The computer-implemented method of claim 7, wherein the augmented view of the physical space includes the selected virtual object overlaid over an image of the physical space.
 16. The computer-implemented method of claim 7, wherein the augmented view of the physical space includes the selected virtual object overlaid over the physical space itself via a translucent display.
 17. One or more non-transitory computer-readable storage media having stored thereon instructions that, upon execution by one or more computing devices, cause the one or more computing devices to perform operations comprising: receiving data representative of a physical space; determining one or more characteristics of the physical space; determining, from a plurality of virtual objects, a group of virtual objects having rule sets that are satisfied by the one or more characteristics of the physical space; receiving an indication of a selected virtual object from the group of virtual objects having rule sets that are satisfied by the one or more characteristics of the physical space; and inserting the selected virtual object into a view of the physical space to create an augmented view of the physical space.
 18. The one or more non-transitory computer-readable storage media of claim 17, wherein one or more rules associated with the selected virtual object include a rule indicating a characteristic of a surface for insertion of the selected virtual object. 